Skip to content

Keep curl teardown from hanging on stalled DNS - #1562

Open
bmehta001 wants to merge 2 commits into
microsoft:mainfrom
bmehta001:curl-nonblocking-dns-teardown
Open

bmehta001 wants to merge 2 commits into
microsoft:mainfrom
bmehta001:curl-nonblocking-dns-teardown

Conversation

@bmehta001

Copy link
Copy Markdown
Contributor

Problem

The curl transport (Linux) could hang the host process at exit when the collector's DNS is unreachable:

  1. curl_easy_perform() doesn't run the progress/xferinfo callback while a name is resolving. An Abort() during a stalled lookup therefore waited for the full CURLOPT_CONNECTTIMEOUT (5s).
  2. curl_easy_cleanup() (curl 7.87–8.19) and curl_multi_cleanup() (curl 8.20+, where the resolver thread pool belongs to the multi handle) join the threaded-resolver thread. That thread stays blocked in getaddrinfo() for the system DNS timeout, which is tens of seconds when DNS is blackholed.

HttpClient_Curl::CancelAllRequests() drains every operation. Hosts that tear telemetry down from static destructors therefore blocked at process exit; ONNX Runtime is one example. I reproduced this with curl 8.21 (the SDK's fetched version) and curl 8.5 (Ubuntu 24.04).

Fix

  • Set CURLOPT_QUICK_EXIT on every easy handle so cleanup doesn't join a stalled resolver thread.
  • On the modern path (runtime libcurl ≥ 7.80, prereq callback), run the transfer on a private multi handle with CURLMOPT_QUICK_EXIT. The loop uses curl_multi_poll in 100ms slices and checks isAborted between them. A completed transfer always reports its real result. The legacy CONNECT_ONLY path is unchanged.
  • Pass both option ids numerically, with static_asserts whenever the headers define them. A runtime libcurl newer than the build headers then still receives the options; this is common with distro upgrades. An older runtime rejects them harmlessly.

Trade-off: a resolver thread abandoned this way keeps running until getaddrinfo() returns. If a library that statically links libcurl is unloaded while such a thread is still blocked, the thread would return into unmapped code. Normal process exit is unaffected.

Test

The new HttpClientCurlDnsStallTests.DestroyingClientDuringStalledResolutionDoesNotHang is Linux-only. It interposes getaddrinfo() for dns-stall.invalid only and forwards every other lookup through dlsym(RTLD_NEXT). It asserts that destroying the client during the stall returns within 3s with one Aborted response.

It skips when the runtime libcurl is older than 7.87, uses a synchronous resolver, or doesn't resolve through getaddrinfo (c-ares). Each run uses its own generation, so --gtest_repeat is safe.

Validation (WSL Ubuntu 24.04, GCC 13)

Run Result
New test without the fix (fetched curl 8.21) Fails after 60,009 ms (teardown waited for the whole stall)
UnitTests --gtest_filter=HttpClientCurl*, system curl 8.5 37 passed
UnitTests --gtest_filter=HttpClientCurl*, MATSDK_CURL_PROVIDER=FETCH (8.21) 37 passed
New test with --gtest_repeat=3, both curl versions 3/3 passed (1–3 ms with 8.5, about 100 ms with 8.21)
Full UnitTests, both curl versions 730 passed
clang++ compile of HttpClientCurlTests.cpp Compiles cleanly

When the collector's DNS is unreachable, teardown of the curl transport
could block host process exit for the system resolver timeout (tens of
seconds). curl_easy_perform() does not run the progress callback while a
name is resolving, so Abort() waited for the 5s connect timeout, and
curl_easy_cleanup()/curl_multi_cleanup() then joined the resolver thread
still blocked in getaddrinfo().

Set CURLOPT_QUICK_EXIT on every handle, and run modern-libcurl transfers
on a private multi handle with CURLMOPT_QUICK_EXIT, polling the abort flag
every 100ms. Option ids are passed numerically so a runtime libcurl newer
than the build headers still receives them. A Linux unit test interposes
getaddrinfo() to stall deterministically; it hangs 60s without this change.

Files changed:
- lib/http/HttpClient_Curl.hpp
- tests/unittests/CMakeLists.txt
- tests/unittests/HttpClientCurlTests.cpp

Co-authored-by: Copilot <[email protected]>
Copilot-Session: ff620832-ec83-4315-b3e0-6f5d825d45e1
@bmehta001
bmehta001 requested a review from a team as a code owner November 19, 2026 00:01
Include the integer type dependency in ICdsFactory.hpp and avoid the racy mkdir implementation in build-runner-tests.py.

Co-authored-by: Copilot <[email protected]>

Copilot-Session: ea7f5226-1c17-476f-abaa-90cfa4b32e3c

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant